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Preface 

On March 11 and 12, 2004, the United States Department of Edueation (ED) 
and the Character Education and Civic Engagement Technical Assistance 
Center (CETAC) ^ hosted a meeting to discuss the challenges in evaluating the 
Partnerships in Charaeter Edueation grant. The meeting goals were to: 

• Receive information regarding the implementation of evaluation 
designs of eurrent Partnerships in Charaeter Edueation grants 

• Diseuss evidenee-based evaluation designs 

• Engage in conversations regarding key issues in conducting 
scientifically based evaluation of Partnerships in Charaeter Edueation 
grants 

• Offer recommendations for staff, grantees, and the field to conduct 
evidence-based evaluations in character education 

• Provide input into evaluation broehure to be published by CETAC. 

Participants included directors of character education projects, project 
evaluators, evaluation experts, and representatives from ED. The agenda is in 
Appendix A and the participant list and session evaluation appear in 
Appendices B and C, respectively. 



The Character Education and Civic Engagement Technical Assistance Center (CETAC) is 
operated through a contract awarded to Caliber Associates. Two subcontractors support 
Caliber, The Association for Supervision and Curriculum Development (ASCD) and Character 
Education Partnership (CEP). 



March 11, 2004 



Welcome and Overview 

Ms. Linda McKay, senior advisor to the deputy undersecretary, ED’s Office of 
Safe and Drug-Free Schools (OSDFS), introduced Ms. Debbie Price, the new 
deputy undersecretary of OSDFS. 

Ms. Price welcomed guests and explained that it is exciting to be working in 
OSDFS because if schools are not safe, a child cannot learn. Additionally, the 
language about character education in No Child Left Behind (NCLB) is 
encouraging because if character education were implemented in every school 
across the country, there would be no reason for OSDFS. 

Ms. Price introduced Dr. Eugene Hickok, the acting deputy secretary of ED, 
who has the monumental task of implementing NCLB. Prior to joining ED, he 
was Pennsylvania’s secretary of education and administered all educational 
programs including K-12 and higher education. Dr. Hickok has authored 
numerous publications and books on history and political science. 

Dr. Hickok spoke about the importance of evaluating character education and 
civic engagement. Under NCLB, evaluations will demonstrate the effect of 
public policy through rigorous research. He stated that both character 
education and civic engagement are difficult issues to promote because of the 
emphasis on accountability and assessment. However, he believes that if, as a 
nation, we do not make sure our children develop character, nothing else 
matters. Having OSDFS speaks to where we are as a nation and says 
something about the need for character education and civic engagement. 

Dr. Hickok’s writings always have focused on the values behind the American 
Republic, and it is important to him that those characteristics are what he 
talks about. He believes that as a nation, we are challenged to help young 
people develop into responsible self-governing citizens and develop matters of 
the heart. 

He believes that the challenge grows greater everyday. Grant competitions help 
create character education programs and it is the grantees’ responsibility to 
make them work. He can imagine no greater responsibility. 

Dr. Hickok told the participants about a visit to an elementary school in inner 
city Baton Rouge that had increased test scores. At a student-held press 
conference, one fifth grader asked, “How can we make sure our schools are 
safe?” Dr. Hickok responded, “There are things we can do. We can have metal 
detectors, cameras, security officers, and we do all that, but if you want to have 
safe schools the most important thing you can do is to look out for each other, 
get to know each other, recognize this is your school and this is your family 




during the day. Look out for your school as you would your home.” This is 
good character. You cannot create good character through public policy you 
can only create the opportunity for good character. In schools and districts 
with character education grants, there are kids the staff will never meet and 
families they will never know whose lives will be better because of the grant. 

Overview of Partnerships in Character Education Program Legislation, 
Program Goals and Objectives 

Ms. McKay gave an overview of the Partnerships in Character Education 
Program and described key goals and objectives for this year. 

The program began in 1994 when Congress authorized funding for 
demonstration grants to States through bipartisan legislation that outlined 
specific character traits that the programs must promote. Parent, community, 
and student involvement were mandated. Other requirements of the law 
included technical assistance for local education agencies and clearinghouses 
in each State. Suggested factors of success included a decrease in discipline 
problems and an increase in student academic performance, participation in 
extracurricular activities, parental and community involvement, faculty and 
administration involvement, and student and staff morale. Forty-five States 
and the District of Columbia received grants. 

NCLB reauthorized the Partnerships in Character Education Program and 
funding increased from $8 million per year to $25 million. Other changes in 
the reauthorization were that both State and local education agencies could 
apply for grants, the character elements were suggested rather than required, 
and character education and civic engagement curriculum would be integrated 
into classroom instruction to align with State academic content standards. 
Allowing local education agencies to apply for the grant was a powerful change 
because local control has more depth of implementation and sustainability. 
Scientifically based research was a competitive priority, and additional funds 
were authorized for national activities in research, dissemination, and 
evaluation. Other key points of the reauthorization were that students with 
disabilities were specifically included and the clearinghouse requirement was 
eliminated. Since 2002, five State and 42 local education agencies received 
grants. 

In 2003, OSDFS was established along with the positions of deputy 
undersecretary and associate undersecretary, and the appointment of a senior 
advisor to the deputy undersecretary for character education. Two new 
positions were created — director of character, civic, and correctional education 
and character education program specialist. In 2003, three integral activities 
occurred which highlighted the importance of character education and civic 
engagement. First, the What Works Clearinghouse (WWC) established 
evidence-based research that led to a definition of character education. 




Second, the Institute of Education Science funded eight social and character 
development grants. Lastly, funding was granted for the ED to develop several 
publications related to character education and civic engagement. (See 
Appendix D for an Overview of the Partnerships and Character Education 
Program) . 

Introduction to the Character Education and Civic Engagement Technical 
Assistance Center 

Dr. William Moore of Caliber Associates provided the overview of CETAC. 
CETAC’s key activities are to provide training and technical assistance for 
grantees, to develop resource materials and publications on character 
education and civic engagement, to develop and maintain a publicly accessible 
Web site, and to engage in meetings and seminars. Training and technical 
assistance are based on grantees’ needs and help to build the grantees’ 
capacity for sustained, effective program implementation and evaluation. 
Publications will support training and technical assistance, disseminate 
current knowledge and state-of-the-art practices, and contribute to the 
knowledge base about effective character education and civic engagement 
processes and practices. The CETAC Web site is a grantee resource and 
learning center and is a vehicle for collaboration and communication. The 
CETAC URL is www.cetac.org . The complete text of the CETAC presentation is 
in Appendix E. 

Ms. McKay thanked Mr. Craig Stanton for the new invitational priority 
language then introduced Dr. Diane Berreth, the meeting facilitator. 

Dr. Berreth is deputy executive director of ASCD and has been involved in 
character education for a decade. She is also a founding member of the CEP. 

Format and Guidelines for Working Together in the Next Two Days 

Dr. Berreth explained that the participants were asked to attend the meeting to 
accomplish the following: 

• Receive information regarding evaluation designs of 2002 
Partnerships in Character Education grants 

• Discuss evidence-based evaluation designs 

• Engage in conversation regarding key issues in conducting 
scientifically-based evaluation of Partnerships in Character Education 
grants. 



Grantee Presentations on Key Issues in Evaluation 

Dr. Berreth introduced the next presenters, each representing a Partnership in 
Character Education grant. Presenters provided an overview of their project, 
evaluation design, and evaluation challenges. 

Jefferson County Public Schools - Ms. Sheila A. Koshewa, Project CARE 
Coordinator, and Dr. Marco Munoz, Evaluator 

Ms. Koshewa stated that the project has two goals: (a) integrating social and 
ethical learning, and (b) creating a school as a supportive environment. Key 
strategies to achieve the goals were selected, most of which focus on 
professional development. Evaluation includes measures to determine learning 
by participants, implementation of learning, and outcomes. Key strategies to 
measure learning and implementation include: 

• Maintaining an attendance sheet for each professional development 
session to measure participation 

• Compiling session evaluation data 

• Conducting classroom walk-throughs to measure level of 
implementation . 

Dr. Munoz discussed the comprehensive primary measures — student and 
teacher questionnaires used to measure goal achievement. The teacher 
questionnaire addresses education beliefs and attitudes, feelings about self as 
a teacher, and school climate. The student questionnaire addresses the areas 
of classroom, school, social, and moral orientation. 

A quasi-experimental design matched the schools and adjusted for difference in 
the baseline data. Schools are matched by characteristics such as 
achievement, socioeconomic status, percentage of exceptional students, and 
mobility. A key strength of the evaluation is the comparison group, which will 
undergo the same measures and matching procedures connecting strategies 
with measures. 

Dr. Munoz noted that their Institutional Review Board (IRB) required both 
parental and student consent since 3rd, 4th, and 5th graders are involved. 
Obtaining the students’ consent was treated as a teaching lesson as evaluators 
showed the students the data collection technology. 

The project staff and evaluator encountered many challenges: 

• Accounting for principal and teacher turnover; especially teachers 
from treatment schools transferring to control schools. 




• Making character education part of the reading program to avoid it 
becoming an add-on. However, this is a compounding factor with 
reading so there will be some concern expressed by teachers. 

• Working with the leadership within the buildings and eontrolling the 
level of implementation. 

Dr. Munoz and Ms. Koshewa addressed a question about whether the parental 
consent forms had to be returned. Dr. Munoz confirmed that the forms did 
have to be returned because the project requires active consent. He 
aeknowledged that they have 60 pereent eonsent, whieh on the primary 
measures, eould be a eonfounding variable, but for the seeondary measures, it 
will be 100 percent. Ms. Koshewa pointed out that the parental consent form 
may seare many families, thus the eonsent form ineluded a letter from the 
prineipal with language that parents would understand. Consent forms were 
included with report cards. 

Uintah School District - Mr. A.J. Pease, Project Director, and 
Dr. Brian Flay and Dr. Don Workman, Evaluators 

Mr. Pease noted that the Uintah projeet is based on the Positive Aetion 
character education model. He stressed that Uintah School District is unique 
because most schools serve only two to three grades and have drastically 
diverse demographie eharaeteristies. Mr. Pease stated that Positive Aetion 
ineludes six elassroom eurrieula units and kits for prineipals, the eommunity, 
and students, and focuses on the sequence of positive feelings, thoughts, and 
aetion. 

Mr. Pease addressed challenges encountered by the project: 

• Contending with a short implementation period. 

• Designing a research study that considered human factors such as 
professionals who applauded research, but crusaded against it. 
Prineipals from eontrol sehools deeided they wanted the program, but 
finally agreed to participate once they understand the research 
design. 

• Contending with ostraeism by the State beeause the State also 
applied for a character education grant and was turned down. 

• Facing a backlash against NCLB because the teachers felt that they 
eould not integrate eharaeter edueation with the foeus on 
aeeountability. (Mr. Pease stated that if we do not deal with eharaeter 
and only address academics, we would be left with smarter crooks). 

• Obtaining IRB approval took momentum away from the study. To 
regain the momentum, project staff had to remind principals of their 
original commitment and that they had project money in their 
budgets. 




Dr. Flay addressed the evaluation design. The school district’s organizational 
structure does not allow controlling for variables between schools, thus grades 
are randomly assigned to each cohort. For example, some schools start with 
odd number grades, others even number grades, and next year the process will 
reverse. Positive Action will not be implemented schoolwide. When the project 
started, staff did not focus on sensitive behaviors and, therefore, obtained 
passive consent but in future data collection, the staff will seek active consent. 
Staff hope for a high level of active consent since families and the school staff 
are familiar with the project and consent forms will be included in the school 
registration packets. Dr. Flay addressed a question regarding whether teachers 
who use Positive Action in the first year will use it the following year. Dr. Flay 
responded that due to the design, they would not because this may cause some 
contamination of results. 

Cooperating School Districts of St. Louis - Ms. Liz Gibbons, Project 
Director, Dr. Sarah D. Caldwell and Dr. Jon C. Marshall, Evaluators 

The Cooperating School Districts (CSD) of St. Louis use CHARACTERplus, a 
character education process coordinated by CSD, parents, businesses, and 
community organizations. CSD is implementing the program in 64 elementary, 
middle, and high schools over a four-year period. Dr. Marshall outlined that 
the key study hypotheses compare changes made by those in treatment 
schools to those in control schools in the following areas: 

• Student character traits, behavior, and achievement levels 

• Parent attitudes about key school climate traits 

• Staff attitudes about key school climate traits 

• Implementation of the key components of the CHARACTERplus 
process 

• Level of district support for implementation of character education in 
schools. 

The project is being implemented in 16 schools each year for a total of 64 
schools. 

Dr. Marshall discussed challenges and barriers: 

• Maintaining school commitment, especially for schools implementing 
in subsequent years. The staff addressed this barrier by letting the 
schools know it is important to collect data even though they are not 
yet receiving the services. 

• Keeping schools on track by having the treatment schools participate 
in all treatment activities and control schools not implement the same 
or similar activities. Video conferencing increased the percentage of 
professionals attending sessions. 




• Obtaining IRB approval to collect data and complete the research. 

The staff obtained passive consent, as opposed to active consent, 
because the large numbers of students would make active consent 
costly and time-intensive. Building trust between the researchers and 
the IRB helped facilitate IRB approval. 

• Educating others about the difference between research and 
evaluation, specifically helping people understand that the project is 
research-driven, and that complete and valid data are critical to the 
project. Educating participants about the value of research and 
helping them understand that their participation will help influence, 
local. State, and national policy about future educational decisions. 

• Linking treatment-specific outcomes to student achievement is 
difficult since State assessments often change, data measured in the 
spring are not available until the fall, data are not available in 
statistically friendly formats, and data might only be available at a 
school level. 

• Principals who collected data on student behavior do not know it is 
easy and rebel against it. Furthermore, IRB requirements of active 
consent to obtain individual scores add to the problem. 

There was some discussion regarding whether this design was a true 
randomized experiment or a quasi-experiment. Dr. Marshall said that he 
understands that an experimental design randomly assigns students to either 
treatment or control schools. Since he is randomly assigning schools, not 
students, he believes it is a quasi-experiment. Dr. Ricky Takai, commissioner 
of the National Center for Education Evaluation and Regional Assistance of 
ED’s Institute of Education Science, joined Dr. Munoz, and Dr. Flay pointing 
out that random assignment and random selection are different and since 
Dr. Marshall randomly selected schools and performed random assignment at 
the school level, they believed the design was a true experiment. The complete 
PowerPoint presentation is in Appendix F. 

Alliance City Schools - Ms. Christine Nardis, Project Director 

Ms. Nardis stated that the project began in 2001-2002, when the district 
received a Department of Health and Human Services planning grant that 
provided the support to conduct a needs assessment of students, teachers, and 
community leaders, elicit community support, and design a program. Using a 
character education grant, the district implemented a program that targeted 
grades K-8 with approximately 2,400 students in six buildings. The numbers 
for the comparison and control groups are very close. The secondary 
population targeted by the grant was 23,100 residents of Alliance, which is a 
microcosm of a big city and has the highest crime rate per capita in the State. 




Outcomes measured by the evaluation included: 

• Student attitudes toward the applications of character and 
perceptions of school 

• Student behaviors 

• Teacher, staff, and administrator perceptions of student character 
and behavior 

• Teacher, staff, and administrator perceptions of the penetration of 
character principles in the school 

• Teacher attitudes toward teaching 

• Community attitudes toward character principles 

• Teacher and staff reports of parent involvement in the schools. 

The evaluation design consists of a longitudinal analysis of: (a) student, 
teacher, staff, administrator, and community attitude change, and (b) student 
behavior change and a quasi-experimental comparison group wait-list design. 

In Year 1, full implementation occurred in two schools and in Year 3 
implementation begins in the remaining two schools. Student data are 
collected each fall and subsequent spring on each new fourth-grade cohort in 
all schools. Each spring thereafter, the survey is repeated as the youth move 
through the 5th, 6th and 7th grades. Baseline survey data were collected from 
the 4th grade in February 2003. Since individual students are not identified or 
tracked, they can use passive consent. 

Evaluation and assessment challenges include: 

• Obtaining a ruling on whether passive consent would be appropriate 
for students in this project. The University of Akron’s IRB approved 
this protocol. 

• Timing of student surveys, because February is the month in which 
the State emphasizes proficiency testing, thus many schools do not 
take time to administer surveys. Many surveys were returned in late 
March after spring break. 

Assessment successes included: 

• Teachers’ support in pre-testing, critiquing survey items, and 
providing continuous feedback each time the survey was administered 
to students 

• Support from the superintendent, who was instrumental in 
cooperating with the passive consent permission letters. 

Ms. Nardis responded to a question about how school culture affects the 
results by saying that project staff were mindful of school culture when 
designing the program. She stressed that it was important to know the 
community and noted that the project has a strong community component. 




Ms. Nardis also answered a question about the extent to which teacher 
feedback impacts student surveys and how that would impact the evaluation 
longitudinally. She responded that the project has changed very little. 
Appendix G contains the complete PowerPoint presentation. 

Orange County Department of Education - Dr. Doug Grove, Evaluator 

Dr. Grove delivered a presentation about why and how the Orange County 
Department of Education established its own IRB; however, he emphasized 
that he was not trying to encourage anyone to set up an IRB. 

When the district was notified about ED funding, they realized IRB approval 
was needed but lacked support from the local university IRB. Project staff did 
not utilize the entire planning year to design strategies and obtain IRB 
approval. As the only evaluator in the office. Dr. Grove sent out 20 letters to 
recruit IRB members and received 12 confirmations. IRB training was 
conducted with the assistance of an attorney. Dr. Grove summarized key 
issues in developing an IRB: 

• Obtaining legal support help 

• Communicating with, and receiving support from, Jeffrey Rodamar, 
protection of human subjects coordinator, at ED 

• Reviewing sample IRB policies and a sample research protocol 

• Obtaining support from the district superintendents. 

One challenge has been replacing IRB members who move and maintaining a 
stable group of alternates. When faced with doing this again. Dr. Grove would 
investigate renting an IRB because the time spent locating and training 
members took away from research. Additionally, he would consider partnering 
with a university, although that presents challenges because the university 
would not assume responsibility for the IRB approval. 

Dr. Grove answered a question about whether the IRB ever achieved 
consensus. He replied that during the first meeting the group established the 
priority of creating a culture and IRB members were motivated because they 
wanted to be part of the research process. Appendix H contains the complete 
PowerPoint presentation. 

Key question 1: What are the key issues and challenges in evaluating 
character education grants? 

Dr. Berreth requested that participants consider key question 1 and Exhibit 1 
presents the participants’ responses. Additionally, Dr. Berreth requested that 
the participants provide more in-depth explanations of several issues and 
challenges. Those responses can be found in subsequent exhibits. 




Exhibit 1 

Evaluation Issues and Challenges 

• Being unprepared for regulatory hurdles, such as the IRB process, after 
investigators wrote the grant and obtained funding (For more in-depth responses 
see Exhibit 2) 

• Collecting baseline characteristics of schools versus characteristics of individual 
students and staff (For more in-depth responses see Exhibit 3) 

• Using active versus passive consent, e.g., bias for obtaining only active consent, 
and strategies for aggressively achieving active consent (For more in-depth 
responses see Exhibit 4) 

• Operationalizing “gold standard” and the language we use when we talk about 
design 

• Contending with changing conditions and threat to the evaluation design of a 
large-scale project (For more in-depth responses see Exhibit 5) 

• Risking data contamination across conditions in various environments (For 
more in-depth responses see Exhibit 5.) 

• Understanding the difference between research and evaluation (For more in- 
depth responses see Exhibit 6.) 

• Implementing a program with fidelity (For more in-depth responses see Exhibit 

7 ) 

• Coping with the demand characteristics of data collection 

• Accounting for attrition characteristics of lost sample and effects 

• Overcoming a lack of staff support. 



Exhibit 2 

Issues and Challenges Regarding Preparation for Regulatory Hurdles 

• Some universities obtain IRB approval prior to submitting the grant. Perhaps the 
Federal government could provide a brief overview of the IRB process, when to 
obtain it, or integrate it into the grant writing and submitting process for project 
directors and school personnel. IRB language intimidates school systems, and 
they do not write grants that are sufficient for an IRB approval. An article in the 
Journal for Character Education addresses this issue. 2 

• To renew the IRB requires researchers to do so eight weeks prior to expiration. If 
it is submitted ahead of time, approval still may not be granted. For example, if 
the name of your project is changed, that is an IRB hurdle, but it really should 
not affect data collection. 

• How does the ED feel about corporate IRBs, which are usually more responsive to 
client needs? Does anyone have experience with them? 

• Are there any statistics about school districts with IRBs? After the grant eligibility 
shifted to local school districts, more districts are considering creating their own 
IRB. 

• Each university has a different IRB process; IRB is a culture at the university level 
but for most school districts it is totally new. Perhaps receiving IRB approval 

should be built into the grant application. 



2 



See Sherblom, S. (2004) Issues in conducting ethical research in character education, Journal of Research in 
Character Education . 1(2). 






Exhibit 3 

Issues and Challenges Regarding Baseline Data and Characteristics 



Baseline characteristics are important when discussing outcome data. Without 
baseline data, a study will not meet the standards for scientifically based research. 
Baseline is a gold standard when collecting individual data differences must be 
addressed. Often treating something as a covariant may not be sufficient. 

The scale in which these grant evaluations must be done is complex and more 
expensive than medical research. How feasible is it working at the school level to 
meet all the standards of rigorous research? Doing anything near gold standard 
with the money allotted for evaluation in these grants is impossible, even though 
this is what ED expects. 

We have students nested in classrooms and classrooms nested in schools. A small 
sample size at the school level will not work because one must check for similarity 
on different levels. 

We do have a large-scale study. Analyzing data by school, statistical power goes 
almost to 0; analyzing data by student often yields a strong statistical power. How 
do researchers balance research design with statistical power? 

There is concern about the credibility of the evidence. The term “gold standard” 
implies that there is a silver standard and the fundamental question is what is the 
silver standard? Random assignment at the school level, and random assignment of 
schools demand a large number of schools and money set aside in the grant. 

The grant application in 2002 had two choices in design while this year there are 
four designs. Evidence-based is mentioned more than 100 times in NCLB. The 
definition was left to the ED which is trying to determine what the term means. An 
evidence-based design needs to show that an intervention is working. 

Attempting to answer questions that assess the impact of random assignment is 
important. Random assignment is possible but it might be difficult at either the 
school or grade level. 

Some States have decision-making councils in each school that might accept the 
project. You have to find one that is willing and then find a comparison school. 

What does it do to implementation when schools have not bought in? 

What is an acceptable effect size? Even something small to researchers might be a 
big difference to a student. It is right to raise the issue of gold standard. Quasi- 
experimental design, used to be respected and acceptable 10-20 years ago because 
it would yield data about impact but now has fallen into disfavor. 

No one should think that randomized trials with schools are impossible. Education 
had moved away from it because many argued that it is impossible, but now we are 
reconsidering it. 

Carrying out a quasi-experimental design (QED) well is at least as hard as doing an 
experimental design. Experience shows that experimental research is more 
accurate than quasi-experimental design. What counts as good QED? Similar 
issues apply to experimental designs (e.g., contamination, attrition). The only 
difference in QED is that it is easier to develop a comparison group than it is to 
convince teachers and superintendents to accept random assignment. 

In some projects, schools were randomly selected and placed before they signed on, 
which was part of the recruitment process. 

A randomization issue in medical research is how to ethically assign someone to the 
non-treatment group. How can you withhold possibly helpful treatment? 





Exhibit 4 

Issues and Challenges Regarding Consent 

• There is a concern that those who did not provide consent were the ones who most 
needed the intervention. Additionally, there are research reports that find those who do 
not give consent are at higher risk. The medical model is driving research but for most 
studies passive consent is fine. ED has not addressed this issue. 

• ED is reluctant to weigh in on passive versus active, which is the responsibility of the 
IRB. There is no checklist and both perspectives make a point for their case. Some 
projects used active consent on advice from ED, however students at risk often were not 
included in the research. Conversely, many students wanted to participate in the 
survey but their parents did not give consent. 

• The common rule specifically requires active consent but the head of the department 
can waive any or all of the common rule. In general, the IRB does not want to take 
risks. 

• Some States have specific items that cannot be asked if using passive consent. 

• Constant communication with principals is vital to obtain active consent. Principals 
want all the curricula on display in the offices for parents. 

• When sending the active consent forms home, teachers play an important role in 
making it a priority for the students. There is high variation of return rates at the 
classroom level which is indicative of the classroom climate. 

• Data collectors can become consent form managers. 



• Reach out directly to parents through school community councils, but there is concern 
that parents not be pressured, in violation of their rights. 




Exhibit 5 

Issues Related to Contamination of Data 



• Sites with randomization within the school can easily result in contamination among 
control groups because in the average school other programs may address character. 
Thus, a teacher in the control group may ask for information and the researchers must 
decide if the information should be available. 

• Teachers, principals, and students change schools or new teachers may integrate some 
elements of character education, or the district may change school boundaries. 

• Vertical teaming is a major issue for teachers and to implement a program they need to 
know why they are doing it. Project directors share what they are doing with teachers 
and parents, and teachers and parents share with others. This discussion is healthy, 
but it is not healthy for evaluation. 

• There is concern for teachers who teach a program one year but not the following year. 
They might not be teaching the formal curriculum, but they may use the knowledge and 
strategies the next year. 

• Some projects stated that no other programs (e.g., DARE) are emphasized during 
involvement with this research. Teachers should continue, not expand, what they are 
doing. It can be tracked with scheduled surveys and, unannounced direct observation 
in program and control classrooms several times a year. 

• Teacher reports do not align with classroom practices. 

• Unannounced and long blocks of observation help but they are expensive. 

• Determining the study design is important. The treatment students receive in the study 
group is different from that received by students in the control group. The evaluator 
and project director should discuss what treatment, if any, the control group is getting. 
The difference in treatment should be as sharp as possible between treatment and 
control. 







Exhibit 5 (cont.) 

Issues Related to Contamination of Data 

• The treatment may manifest itself differently in how students of different ages respond 
to the same survey. In a longitudinal study, there may be an effect as a result of the 
students’ developmental progress. Third graders taking questionnaires and their ability 
to test on things that are relevant to character education is worrisome; it might be 
better to start with 4th grade. 

• Contamination is one way designs break down; even randomized designs can break 
down as the structure of schools can change. Tom Cook has written about how to 
guard against contamination and provides useful suggestions such as collecting 
baseline data. Using a quasi-experimental design with randomized controlled trials will 
help maintain the validity of the research. 

• Staff mobility is an indicator of change. 

• In longitudinal studies, receptivity to research increases over time; at first students take 

the survey as if it were a test and become more casual in subsequent surveys but those 
changes are never detected. 




Exhibit 6 

Responses to Research versus Evaluation 



• From the regulatory standpoint, doing something in an unsystematic manner does not 
count as research under the common rule. 

• The same standards are applied to the design. However, the question is, “Are the 
standards the same and should they be? 



• The biggest difference is that practitioners are not trained to make recommendations to 
a superintendent. The audiences, style, and language are different. 




Exhibit 7 

Responses to Implementation of Program Fidelity 



• Some evaluators and project directors have proactive conflict because the evaluator 
addresses implementation issues. The evaluator should be involved in the grant writing 
process. The evaluation report should address operational definitions training of data 
collectors and instructions provided to teachers. The problem is the rigidity of the 
research with the desired flexibility of the program. 

• Using an external evaluator on an in-depth project was new to the schools. What were 
the challenges and solutions? 

- A project leader who is not familiar with evaluation but must report to a higher 
authority in the school makes it difficult to implement an evaluation or the 
research. 

- This is an issue that occurs in all evaluations because school-based professionals’ 
perceptions and culture are different. Evaluator should assume the role of 
providing an orientation about evaluation. Evaluators use a gold standard to judge 
a program’s effectiveness but others do not share that view. 

- The researcher or evaluator needs to be integral from the beginning. One approach 
is that the evaluator is part of the team, not external to the team. 

- Researchers work better when they are knowledgeable about the program and its 
people, and those people understand the researcher’s role as professionals 

complement each other. 









Discussion of Complex Issues in Evaluation Design 



Dr. Moore presented a session on Issues in Complex Evaluation Design and 
stressed that research efforts have not made scientific contributions to 
resolving pressing education problems, operationalizing scientifically based 
research, or identifying appropriate evaluation methodologies. Dr. Moore cited 
a 2001 report by Boruch, De Moya, and Snyder that stated of 84 evaluations 
and studies supported by the ED in FY2000, there was one randomized 
controlled trial, 51 needs assessments, 49 program implementation or 
monitoring evaluations, and 15 non-randomized impact evaluations. 

Dr. Moore reviewed the designs consistent with scientifically based research: 

• Randomized controlled trials 

• Quasi-experimental designs with carefully matched comparison 
conditions 

• Regression discontinuity with repeated measures before and after 
intervention 

• Single-subject such as multiple baseline, treatment reversal, or 
interrupted time series. 

The most challenging issues regarding scientifically based research are 
definitional, philosophical, and pragmatic issues. Dr. Moore concluded with 
the question, “Why has education not adopted a more experimental approach 
to knowledge generation?” Dr. Moore’s PowerPoint presentation is in Appendix 
I. The points raised by participants to Dr. Moore’s presentation are in 
Exhibit 8. 



Exhibit 8 

Complex Issues in Evaluation Design 

• Obtain support from the teachers unions so teachers can complete the surveys and 
participate in the research. 

• Opportunities for correlational studies as building blocks are lost. We need to examine 
the status of the character education knowledge base, which ranges from anecdotal 
evidence to randomized controlled trials. Is it time to look at outcomes and impact? 

• There is a question about how long it takes to see student outcomes thus how early we 
should begin character education with children. By the time they reach kindergarten, 
character is established. 

• Are there common logic models for character education? If there are several logic 
models, are there commonalities? 

• To what extent has character education looked to related fields for outcome measures? 
Character education draws a lot from prevention research. 

• Are there actually added costs to conducting randomized controlled trials? 

• Goals not established at the outset of a study can product research that is not useable 

or valid. Operationalizing character at the start of the project is essential. 






Exhibit 8 (cont.) 

Complex Issues in Evaluation Design 

• What is the audience for the publication? Many project directors have education and 
not evaluation backgrounds. Is it the academics or the project directors who need to 
apply the findings? 

- Researchers do not understand that educators are not concerned about outcomes, 
other than those that focus on academic achievement. 

- Researchers need to examine outcomes to determine if funding makes a difference. 

• Process outcomes are helpful to schools and require collecting different types of data. 
Does the ED view that both process evaluations and research are important? 

• The Institute of Education Science (lES) is struggling with process research. A mandate 
was set, but now we are reviewing how the field is coping with the mandate. We should 
not back away from the high standards but we need to hear from program directors 
what it is like in a school to make research realistic. The practitioner needs to meet the 
theory. 

- lES is focusing on the four designs but there are components within ED that are 
focusing on implementation and process studies. 

• Any good evaluation includes measures of implementation and process which is the 
difference between efficacy and effectiveness studies. 

• During the 1960s and 1970s, the focus moved from randomized controlled trials to 
program evaluation. Traditional research does not allow for modification whereas in 
program evaluation the opposite is true, but the result is no fidelity of data. Perhaps 
articulating the distinction between types of research is needed, each with legitimacy, 
and funded accordingly. 

• Investigating the underlying processes and measure also is important. Who will fund it 
all? 



March 12, 2004 

Review of March 11, 2004 Proceedings 

Ms. McKay thanked everyone for coming and restated the goals of the meeting 
then introduced Mr. Paul Kesner, director, character, civic and correctional 
Education, OSDFS, who introduced Mr. Bill Modzeleski, associate deputy 
undersecretary, OSDFS. Dr. Takai discussed the various evaluations 
standards. 

Legal Issues 

Ms. Ellen Campbell, deputy director. Family Policy Compliance Office, ED and 
Mr. Jeffrey Rodamar, protection of human subjects coordinator. Office of Chief 
Financial Officer, ED discussed legal issues related to conducting research. 

Ms. Campbell talked about the two laws her office administers: the Family 
Educational Rights and Privacy Act (FERPA) which deals with education 
records, and the Protection of Pupil Rights Amendment (PPRA), the parents’ 
rights amendment. It is important that grantees are aware of the information 
needed on a student and share that with school officials. Previously, active 
consent was required if a child was required to take a survey. It is important to 






make clear that all surveys are voluntary, not only to the child, but also to the 
parent. New requirements stipulate what must be sent to parents regarding 
surveys^. The new amendment to PPRA requires that schools provide parents 
an opportunity to review the survey, even for passive consent, which is 
acceptable if handled properly. For example, if a survey asks questions in eight 
specific areas, active consent is required. One such survey item for character 
education might ask if the student has ever done something he or she should 
not have: a survey containing that item requires active consent. Mr. 

Modzeleski remarked that he and Ms. Campbell have discussed the issue of 
voluntary participation, especially in studies involving drugs. He recommends 
erring on the side of caution. 

Dr. Workman pointed out that under the common rule, the idea of consent is a 
misnomer; one either has consent or a waiver of consent. The common rule 
and PPRA are largely overlapping but where there is no overlap, one should 
follow the stricter rule. Participant reactions to this discussion included: 

• These changes reinforced actions taken by IRBs such as explicitly 
stating in the letter to parents that the survey was voluntary, training 
data collector how to communicate with students that the survey is 
voluntary, and avoiding embarrassment among students who do not 
take the survey. 

• There are obstacles to collecting district disciplinary referrals data 
because district officials felt that such data was too defined and that 
it was possible to identify the student, which is especially important 
in this era of accountability. Possible solutions for overcoming this 
challenge were: 

- Using data collectors not affiliated with the school or district 

- Placing teacher surveys in sealed envelopes and assuring teachers 
that no one in the school will see the responses. 

• Principals who are not willing to share information also have teachers 
who are reluctant to provide information connected to performance. 

• Students feel that ever3dhing they do in school is a requirement, thus 
they are reluctant to believe that a survey is voluntary. Possible 
solutions include: 

- Provide students who do not want to participate in the survey 
another activity, give them the opportunity to leave the room, or go 
to the back of the room 

- Instruct data collectors to have the teacher remain in the room, sit 
at a desk or in the front in the corner, and not walk around the 
room. Students are then assured that data collectors do not look 
at their responses, avoiding intimidation. 



^ For more information see www.ed.gov/policv/gen/guid/fpco/htopics/idex.html?expo=0 



- Ask data collectors to take a test for IRB certification, which is 
required by some IRBs. The certification helps data collectors 
understand the requirements for human subjects research. It is 
not clear when or if it will be a requirement for all grantees. 

Key question 2: What recommendations can be offered to help 
accomplish evaluation requirements for Partnerships in Character 
Education grants? 

Dr. Berreth asked the participants for suggestions about evaluating 
Partnerships in Character Education grants. The suggestions provided are in 
Exhibit 9. 



Exhibit 9 

Suggestions For Evaluating Partnerships in 
Character Education Grants 

1. The RFP should: 

• Mandate that projects use a highly qualified evaluator or researcher who 
understands how to conduct the research requested by the department. 

Perhaps institute a planning period in which project staff develop a partnership 
with an evaluator. 

• Continue to recommend external independent evaluator to help the department 
build character education knowledge based on rigorous research. 

• Use evidence-based literature and research in writing the proposal. 

• Discuss the importance of a mixed-method design to evaluate project 
implementation and the use of qualitative data to help interpret quantitative 
data. 

• Explain how gold standard research will be a competitive priority. 

• Emphasize that budgets should be appropriate for the type of research or 
evaluation required by the department. 

• Suggest that the researchers and evaluators be involved in developing and 
writing the proposal. 

• Clarify the current language that addresses involving the independent evaluator 
in decision-making for the project. For example, at what point, how early, and 
where can evaluator be involved? 

• Clarify if the priority is not to do rigorous research and doing so does not give 
competitive advantage. 

• Clarify what should be done during the planning period, e.g., what are the 
objectives of the planning year. 

• Strengthen the RFP by adding deliverables as part of the annual report 
requirement. 

• Reaffirm the use of different evaluation methodologies, and caution against 
assuming the medical model of treatment and evaluation because education is 
different than the doctor and patient relationship. 

2. Develop online training modules about FERPA and PPRA requirements, how to 
apply for IRB clearance, and provide guidance about when character education is 
part of normal education practices (which is important under the common rule for 
exempt research). 

3. Establish a central IRB that would not substitute for local IRB approval, but to help 
grantees document their processes. 

4. Support more programmatic research of character education. 






Exhibit 9 (cont.) 

Suggestions For Evaluating Partnerships in 
Character Education Grants 

5. Revisit the definition of character education, what role it has in public education, 
and identify the knowledge gaps that prevent the field from moving forward. 

6. Commission a theoretical paper, written by both academics and practitioners, that 
will outline the history of character education and broad theories, and their 
implications for defining character education. 

7. Consider balancing the monetary requirements for research with those of the 
program. 

8. Design a logic model that connects accountability in reading, writing, and 
mathematics with outcomes for character education. 

9. Help practitioners answer questions about how to begin developing a relationship 
with an evaluator. 

10. Provide first-year grantees additional support through conferences or monthly 
telephone calls to share tools and discuss issues. 

11. Provide IRB training for new grantees; the concept of gold standard research and 
outside independent evaluators are in conflict. 

12. Develop Web-based research to practice information for parents and administrators. 

13. Help clarify the differences in core purposes perceived by State education agencies 

and parents. 



Ms. McKay introduced Ms. Donna Muldrew, liaison to the First Lady from ED, 
who is working with teacher groups on NCLB. 

The final part of the evaluation sessions involved the participants sharing their 
thoughts about what should be included in a CETAC evaluation publication. 
Participants were divided into the two groups — district-level professionals and 
those not affiliated with a grant, and evaluators. The responses are in 
Exhibit 10. 



Exhibit 10 

Participant Suggestions for the CETAC Evaluation Publication 

District-level Professionals and 

Those Not Affiliated with a Grant Evaluators 

• Reflect what practitioners and project • Define the audience for the publication as 

directors need to know when talking to grantees and new grantees 

teachers and parents • Include indicators and measures 

• Write in laymen's language • Provide and summarize resources and link 

• Define terms such as research-based, for new and potential grantees 

character education, and randomization • Develop a “where to begin” manual with 

• Discuss the benefits (e.g., data-driven worksheets to create ideas 

decision-making, advance policy, inform • Include a glossary 

instruction and practice) an evaluation will • Explain and illustrate logic models 

have for the greater community • Provide a flow chart with a sequence of 

• Discuss constructs for measuring steps once the grantee obtains a grant 

character education outcomes • Provide templates for permission letters 








Exhibit 10 (cont.) 

Participant Suggestions for the CETAC Evaluation Publication 


District-level Professionals and 
Those Not Affiliated with a Grant 


Evaluators 


• Provide illustrative character education 
cognitive and academic outcome indicators 

• Describe what requests evaluators will 
have for practitioners 

• Explain the four methodological designs 
and what each one entails for the 
practitioner 

• Provide a suggested budget (e.g., 
evaluation should involve 10-50% of the 
budget) 

• Provide an overview of the IRB process and 
give guidelines 

• Explain when, how, and where to find an 
evaluator 

• Discuss evaluator's role and 
responsibilities (e.g., member of a team) 

• Provide suggestions for reporting and 
disseminating results 

• Provide examples of letters and permission 
forms. 


• Help readers focus on and use assests and 
resources within the project, the 
classroom, building, district, and 
community at large 

• Provide suggestions about how to cope 
with negative or unexpected results 

• Explain the role and responsibilities of an 
external evaluator and why it is important 
to develop trust. 



Summary and Next Steps 

Dr. Berreth and Ms. McKay expressed their appreeiation to everyone for their 
openness and willingness to trust. Mr. Kesner also thanked the participants 
for taking time to attend the meeting, and noted that the ED and CETAC staffs 
value their eontributions. He quoted T.S. Elliot: “We will not eease from our 
exploration until we have come back to where we started and recognize that 
place.” 
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U.S. Department of Education 

Character Education and Civic Engagement Technical Assistance Center 



Setting the north star for 
our children to live and learn 



AGENDA 



Grant Evaluation Meeting 
Hilton Crystal City 
Washington DC 
March 11-12, 2004 



Meeting Outcomes: 



1 . Receive information regarding evaluation designs 
of current Partnerships in Character Education 
grants 

2. Discuss evidence-based evaluation designs 

3. Engage conversation regarding key issues in 
conducting scientifically-based evaluation in 
Partnerships in Character Education grants 

4. Offer recommendations for staff, grantees, and the 
field to help accomplish evidence-based evaluation 
in character education 

5. Provide input into evaluation brochure to be 
published by CETAC 



Thursday - March 11™ 



Rappahannock Room 

1:00-2:15 Welcome, Celebration and Overview 



1:00-1:15 Ms. Debbie Price - Deputy Under Secretary 

U.S. Department of Education- Office of Safe and 
Drug-Free Schools 

1:15-1:30 Dr. Eugene Hickok 

Acting Deputy Secretary - U.S. Department of 
Education 



Thursday - Cont’d 



1:30-145 Overview of Partnerships in Character Education 

Program legislation, Program Goals and Objectives 

Ms. Linda McKay 

Senior Advisor to the Deputy Under Secretary 
OSDFS - Character Education Civic Engagement 
U.S. Department of Education 



1:45-2:00 Introduction to the Character Education and Civic 

Engagement Technical Assistance Center 

Dr. William Moore 
Caliber Associates 



2:00 -2: 15 Format and Guidelines for our Work 

Dr. Diane Berreth, Facilitator 

Association for Supervision and Curriculum 

Development 

2:15-3 :30 Grantee Presentations on Key Issues in Evaluation 

Ms. Christine Nardis, Alliance City Schools and Dr. 
Peter Leahy, University of Akron 
Ms. Liz Gibbons, CHARACTERplus, CSD, St. Louis, 
MO, Dr. Sarah D. Caldwell, Evaluator and Dr. 
Jon Marshall, Evaluator 

Ms. Sheila Koshewa and Dr. Marco Munoz, Jefferson 
County Public Schools 

Dr. A. J. Pease, Uintah School District, Dr. Brian Flay 
and Dr. Don Workman, UIC 
Dr. Doug Grove, Orange County Department of 
Education 



3:30 - 3:45 Break 



3:45-5:30 Key Questions: What are key issues in evaluations 

of these grants? What challenges have been faced 
in meeting the evaluation requirements? 

Dr. Diane Berreth, Facilitator 
All 





Thursday - 


Cont’d I 


5:30-6:15 


Extended Break 


6:15-7:45 


Dinner and Discussion of Complex Issues in 
Evaluation Design 

Dr. William Moore 


7:45-8:00 


Framing Tomorrow’s Work 

Dr. Diane Berreth, Facilitator 
Ms. Linda McKay 


Friday - March 12th | 


Rappahannock Room 


9:00-9:15 


Recap of Day 1 

Dr. Diane Berreth, Dr. William Moore, 
Ms. Linda McKay 


9:15-9:45 


Evaluating the Effectiveness of Character 
Education Programs/ Q&A 

Dr. Ricky Takai, U.S. Department of Education 


9:45-11:00 


Reflections and Key Question: What constitutes a 
strong evaluation design vision for character 
education programs? 

Dr. Diane Berreth, Facilitator 
All 


11:00-11:15 


Break 







Friday, Cont'd. 



11:15-12:30 


Key Question: What recommendations can be 
offered to help accomplish the evaluation 
requirements for Partnerships in Character 
Education grants? 

Dr. Diane Berreth, Facilitator 
All 


12:30-1:30 


Lunch 


1:30-2:45 


Discussion: Based on our discussion yesterday and 
today, what should be included in a CETAC 
publication focused on evaluation? 

Dr. Diane Berreth, Facilitator 
All 


2:45-3:00 


Summary and Next Steps 

Ms. Linda McKay Mr. Paul Kesner U. S. Department 

of Education 

Dr. William Moore 
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our children to live and learn 


U.S. Department of Education 

Character Education and Civic Engagement Technicai Assistance Center 




Feedback Form 






CETAC Evaluation Meeting 
March 11-12, 2004 
(n=13) 




Please eomplete the following feedbaek form so that we may determine the 
effeetiveness of this meeting and better prepare for future meetings. 


1. Please indieate the extent to whieh the following meeting objeetives were 
achieved. Use the following seale: l=not at all; 2=minimallv: 3=mostlv: 
4=fully): 


a. Received information regarding evaluation 
designs of current Partnerships in Character 
Education grants. 

Comments: 


3.5 (average) 


b. Discussed evidence-based evaluation designs. 
Comments: 


3.6 


c. Engaged in conversation regarding key issues in 
conducting scientifically-based research in 
Partnerships in Character Education grants. 
Comments: 


3.9 


d. Offered recommendations for staff, grantees, 
and the field to help accomplish evidence-based 
evaluation in character education. 

Comments: 


3.8 


e. Provided input into the evaluation brochure to be 
published by CETAC. 

Comments: 


4 



2. Please rate the quality of the following 
2= satisfactory; 3=good; 4=excellent): 


on a scale of 1 to 4 (l=poor; 


a. Participant folders/ resource materials: 
Comments: 


3.7 


b. Meeting organization: 
Comments: 


4 


c. Key questions: 

Comments: 

■ Focused, well-passed. 


3.9 


d. Hotel accommodations and overall 
meeting room facilities (e.g., room size, 
atmosphere, etc.): 

Comments: 


meals 

3.7 


e. Overall meeting: 
Comments: 


4 


3. Please rate the preparedness of the facilitator on the following 

aspects using a scale of 1 to 4 (l=poor; 2=satisfactory, 3=good; 4=excellent): 


a. Knowledge about the content of 
the meeting: 


3.8 


b. Background and experience related 
to the content of the meeting: 


3.6 


c. Ability to facilitate meeting: 


3.9 


d. Overall rating of the facilitator: 


3.9 



Comments: 

■ Outstanding job! 

■ Excellent facilitation with a tough group. 

■ Some comments edged close to being “put-downs.” 

4. Which discussion(s) was most useful to you? 

■ Key issues in evidence-based research. 

■ Research design discussions of contamination. 

■ Design evaluation. 

■ Philosophical Issues. 

■ IRB. 

■ Discussion on recommendations, when each of the participants had 
2 minutes to speak. 




5. What other information would have been useful to you for this meeting? 

■ Analysis of four basic designs associated with RFP. 

■ Information on human subjects protection requirements. 

■ Review of the good research in character education. 

6. What recommendations do you have for future meetings? 

■ Talk for about specifics related to randomized controlled trials. 

■ Discussion of the key literature findings in character education. 

■ In depth discussion of design options, designs, technical 
approaches. Let experts debate issues and come to consensus. 
More fucis on terminology and evaluation purpose(s). Often tended 
to get too broad trying to satisfy too many needs. 

■ Theory paper - character education as priority. 

■ Have more program people. 

7. Additional comments: 

■ Pleased to see the shared challended of researchers/ practioners - 
we are not alone. 

■ Unclear whether these discussions will have influence. Feels four 
designs are written in stone. 
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U.S. Department of 
Education 




Partnerships in Character 
Education Program 




PCEP 



• Funding Authorized 
1994 



by Congress 




• Pilot Demonstration Grants to 
implement character education 

• Approximately $8 million per year 



PCEP Legislation (94) 



Only State Educational Agencies 
eligible 

Specified elements of character 
45 states and DC received awards 
Involve parents, students & community 
Curriculum & instruction practice 



PCEP Legislation (94) 



Technical Assistance to LEAs 



Establish a Clearinghouse 
Teacher training and parent education 





PCEP LegisI 



Factors of success 



- Decrease in: 

• discipline problems 



( 94 ) 




PCEP Legislation (94) 



Factors of success: 



- Increase in: 

• student academic performance 

• participation in extracurricular activities 

• parental & community involvement 

• faculty & administration involvement 

• student & staff morale 



PCEP Reauthorization 



No Child Left Behind Reauthorize 

Funding Increased to $25 million 
year 




PCEP Reauthorization 



KEY POINTS OF REAUTHORIZATION 

• State & Local Educational Agencies" 

• Character elements suggested 

• Integrated into classroom instruction 

• State academic content standards 



PCEP Reauthorization 



KEY POINTS OF REAUTHORIZATION 

• Educational reform 

• Scientifically based research 

• Students with disabilities specifically 
included 

• Eliminated clearinghouse requirement 



PCEP Reauthorization 




Success factors added: 

- Student academic achievement 

- Improved school climate 



PCEP 



Since 2002 

• 5 SEAs 

• 42 LEAS 



authorization 




PCEP Reauthorization 



Established a set aside for national 
activities in research, disseminatio 
evaluation 



2003 Events 



Developed plan to support the U.S. 
Department of Education’s Strategi 
Objective 3.2 — ’’Promote strong 
character and citizenship among ourfm 
nation’s youth” 




2003 Events 



• Establishment of the Office of Safe 
Drug-Free Schools (OSDFS) 

• Established positions of Deputy Under 
Secretary and Associate Deputy Under 
Secretary for OSDFS 

• Appointment of Senior Advisor to the 
Deputy Under Secretary for Character 
Education 




2003 Events 



Director of Character, Civic & 
Correctional Education Programs 

Hired Character Education Progra 
Specialist 




2003 Plan 



What Works Clearinghouse evidence- / 
based study of Character EducatioB^ 
- 1 study of the Clearinghouse 



2003 Plan 



Support of Institute of Education / 
Sciences 

- 8 Social & Character Development Grants 



PCEP FY 2004 



2004 grant package is now available 

Nearly $2.5 million available for 
awards 

Plan to make between 6-9 new awards 
Application Due Date: 9 April 2004 



Publications 



Establish the written word for Character 
Education 

ED publications on Character Education 

- Review of state pilot grant reports 

- Evaluation to guide the field 

- Key topics and issues in character education 

- Fact sheets relating CE to NCLB 



CETAC 



• Establishment of Center for Character 
Education & Civic Engagement 

• Partners 

- US Dept of Education — OSDFS 

- Caliber Associates, Inc. 

- Association for Supervision and Curriculum 
Development (ASCD) 

- Character Education Partnership (CEP) 



CETAC 



Key Activities of CETAC / 

- Provide training and technical assistdn^ ^ 

for grantees ^ ^ 

- Develop resource materials and publication 
on CE 

- Develop and maintain a publicly accessible 
Website 

- Meetings and seminars 



Setting the North Star for 
children to live & learn 
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Introduction to the Character Education 
and Civic Engagement Technical Assistance 

Center (CETAC) 




Introduction to the Character 
Education & Civic Engagement 
Technical Assistance Center 



U. S. Department of Education, 
Office of Safe and Drug Free 
Schools 




Caliber Associates, Inc. 

Association for Supervision and Curriculum Development 
Character Education Partnership 
March, 2004 



CETAC Partners 



• US Department of Education, Office of/ 
Safe and Drug Free Schools 

• Caliber Associates, Inc. 

• Association for Supervision and 
Curriculum Development (ASCD) 

• Character Education Partnership (CEP) 




• August 6, 2003... RFP issued by the Department of Education 
to create and implement a character education and civic 
engagement technical assistance center 



• September 29, 2003. . .Caliber and its partner ASCD a^ard 
contract to design and implement the new center 

• October 13, 2003... Character Education & Civic Engagement 
Technical Assistance Center Kick-Off 

• October 16, 2003... Secretary Paige announced establishment 
of CETAC at CEP 10* anniversary dinner 

• December 12, 2003. ..CETAC introduced to grantees 

• January, 14, 2004.... CETAC convenes Center Resource Group 

• February 1 0, 2004. . . .CEP joins Caliber and ASCD as partners 
in CETAC deliverables 




Rationale for CETAC 



• Increased support to grantees 

• Information to the field and need for ED 
publications 

• Fieid evoiution and expansion 

• Heaithy debate about CE processes and 
associated outcomes 

• Stage of deveiopment warrants center 



Important Questions Need Clear 
Answers: The Work of CETAC 



What is character education and why is it 
important to our nation’s youth? 

What approaches are most successful for 
students of different ages, interests, needs and 
backgrounds? M 

What outcomes can be realistically expected to 
result from CE processes and programs? 





• What staff development activities and working 
conditions are necessary to promote effective 
CE program implementation? 



Grantee Needs Guide the Work 
of the Center 



• Surfacing the challenges through 



• Identifying and selecting core components ot 
effective models 

• Successfully implementing and sustaining 
proven practices in character education 

• Effective evaluation of the implementation and 
impact of CE programs 



needs assessment process 
• Known needs 




Purpose of CETAC 



• To provide grantee administrators with 
information on: 

- effective grant implementation practices (around^ 
program implementation and evaluation) in 
character education, and 

- effective CE processes, program componen ts an d 
instructional strategies that support and 
compliment existing reform efforts, academic and 
youth development outcomes 

• To serve as a central resource for the broader 
education community 

• To foster high quality implementation and 
evaluation practices but not to impose or select 
CE programs 



Key Activities of CETAC 



• Provide training and technical 
assistance for grantees 

• Develop resource materials and 
publications on character education 

• Develop and maintain a publicly 
accessible website 

• Meetings and seminars 



Training and Technical Assistance 
Services 



Guided by grantee needs 

Designed to build grantee capacity for 
sustained effective programming 

Focused on the primary grantee activities of 
program implementation and evaluation 

Tailored to grantee needs using a mix of 
different technical assistance methods 




Publications 



• Intent: 

- To support training/TA needs 

- To disseminate current knowledge and stat 
the-art practices 

- To contribute to the knowledge base about 
effective CE processes and practices 

• Types: 

- Training-specific documents 

- Fact Sheets 

- Brochures 

- Review of state pilot grant reports 

- Monograph 



CETAC On-Line Website 



Serves three fundamental purposes 

- Grantee resource and learning center / 

- Information dissemination 

- Coiiaboration and communication vehicie 

. . . and has four key features: 

• CETAC Resource Center 

• News and Events 

• Grantee and CRG Work Zone 

• Site Tools 

• www.cetac.org 



CETAC/ 

uur ^Jtatv^rr to /tve ontf farm 



U.S. Department of Education 

Character Education & Civic Ennagement Technical Assistance Center 

CETAC Online 




Vou At-e HerciCETAC Home 



Home 




Welcome to the 
Character Education 
and Civic Engagement 
Technical Assistance 
Center (CETAC)T 



Welcome to the Character Education and Civic Engagement 
Technical Assistance Center (CETAC) website, CETAC Online! 
Sponsored by the U.S. Department of Education's (ED) Office of 
Safe and Drug Free Schools (OSDFS), CETAC Online provides 
State program administrators, local educators, and the public 
with information on character education and civic engagement, 
as well as strategies that support academic goals and other 
reform efforts. CETAC Online contains information for users on 
programs, legislative changes, current research, news and 
events. CETAC's objective is to provide technical assistance on 
implementing effective character education and civic 
engagement to U.S. Department of Education-Partnerships in 
Character Education Program grantees and to serve as a 
resource for grantees, educators, parents, and the community 
at large. 

To find out more about the Partnerships in Character Education 
program, click here. 



^ Contact CETAC Staif 



Have a question, comnnent or suggestion 
for the CETAC staff? 



Bnt CETAC New* And EwniB 



the short tile oF the news - Jan 9, 2004 
this is a test description. 

CETAC V^/ebsite Announces - Dec 16, 2003 
The CETAC Website is live now! 

Q5DF5 Announces - Dec IS, 2003 

The Character Education and Civic Engagement Technical Assistance 
Center is established and sponsored by the department oF Education's 
office of Safe and Drug Free School. 

CETAC First Publication - Dec IS, 2003 

CETAC First Publication has been Posted on the CETAC Website. The 
was generated based on the study on 100 middle schools in the northeast 
area of US. 

5 entries | More News Si Events | Event Calendar 






Keyword, Publication, Etc. 
More Search Options 



Work ZiM« Onliiw 






User Id : \ 
Password: |~ 



About Work Zone 
Forgot Your Password? 




Accessibility I Privacy Policy and Disclaimer I Security I Back I Toi 
Last Updated: January 12, 2004 





Meetings and Seminars 




• Meetings for ED and CETAC staff to 
identify the key topics and issues in ^ 
character education... and specifically in 
effective CE processes, implementation 
and evaluation 

• Seminars for ED staff, character 
education and national education 
organization representatives 



Meetings and Seminars 



CRG Meeting — Jan 14-15, 2004 
National Listening Session — March 29-30, 2004 
CETAC Evaiuation Meeting — March 11-12, 2004 
Speaker’s Sessions for ED staff — Kickoff April 1 , 




2004 
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Missouri CHARACTERpZus® 
Implementation Project 




Missouri CHARACTERp/u^ 
Implementation Project 



Director 

Liz Gibbons iqibbons@csd.edu 



Researchers 

Jon C. Marshaii ioncmarshaii@aoi.com 

Sarah D. Caidweii sdcaidweii@aoi.com 



Coordinator 

Jeanne Foster ifoster@csd.edu 



See showmecharacter.org web site for more information 



August 9, 2004 



CHARACTERplus Implementation Project 



1 



Key Hypotheses (Research Questions) 



■ Students in treatment schools will demonstrate 
statistically significant (p < .05) positive change in key 
character traits compared to students in controi schools. 

■ Students in treatment schools will demonstrate 
statistically significant (p < .05) positive change in 
behavior as compared to students in control schools. 

■ Students in treatment schools will demonstrate 
statistically significant (p < .05) positive change in 
achievement levels than students in comparison schools 
when other factors are controlled. 



August 9, 2004 



CHARACTERplus Implementation Project 



2 



Key Hypotheses (Continued) 



■ Parents in treatment schools will demonstrate statistically 
significant (p< .05) positive change in key school climate traits as 
compared to parents in control schools. 

■ Staff in treatment schools will demonstrate statistically significant 
(p< .05) positive change in key school climate traits as compared 
to parents in control schools. 

■ Treatment schools will demonstrate statistically significant (p < 
.05) positive change in implementation of the key components of 
the CHARACTERp/as program as compared to control schools. 

■ Treatment schools will obtain high levels of district support for 
implementation of character education as compared to control 
schools. 



August 9, 2004 



CHARACTERplus Implementation Project 



3 



Focus - Experimental Implementation of 
Character Education Project in 64 Schools 



■ Project Components [Treatmenf\ 

□ CHARACTERp/us Ten Essentials | 

□ PIR Databased Planning * 



Integrated 

program 



□ 



□ 

□ 



□ 



Parent, Staff, Student, Implementation 
Assessment in February 

Student Behavior Tracking (continuous) 

June Training [Content and Databased 
Process] 

Networking Sessions During the Year 



Surveys 



Referral 

Form 



Training 



□ Continuous Coaching During the Year 



August 9, 2004 



CHARACTERplus Implementation Project 



4 



Model 



/ 



four groups: 

4 

4 

4 

4 



■ 4 schools from each group (16 total) randomly placed 
to begin treatment each year 

■ Extra schools were randomly selected as alternates to 
substitute for districts or schools choosing not to 
participate 



Random sample of schools stratified into 
16 Elementary Schools 
16 Junior High or Middle Schools 
16 Senior High Schools 
16 Secondary Schools 



□ 



□ 



□ 



August 9, 2004 



CHARACTERplus Implementation Project 
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Design 



Selection & 
Placement 


FY2003 


FY2004 


FY2005 


FY2006 


Random 


0 


T1 


0 


T2 


0 


T3 


0 T4 


Random 


0 


C1 


0 


T1 


0 


T2 


0 T3 


Random 


0 


C1 


0 


C2 


0 


T1 


0 T2 


Random 


0 


C1 


0 


C2 


0 


C3 


0 T1 



0 - means survey data collected February of each year 
T - means treatment year 
C - control group year 



August 9, 2004 



CHARACTERplus Implementation Project 
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B Sampling 



■ stratified random sample of schoois 

■ Districts stratified by socioeconomic ievei and 
size based on previousiy published study of the 
state 

■ Random sampies drawn by district to represent 
both high and iow socioeconomic leveis (oniy 
for representation) 

■ Randomly selected two to four schools per 
district depending on the size of the district 



August 9, 2004 



CHARACTERplus Implementation Project 
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Number of Surveys Collected First Year 



Survey 


T2003 


T2004 


T2005 


T2006 


Total 


Parent [26% return] 


276 


343 


183 


327 


1,128 


Staff 


313 


339 


500 


407 


1,559 


Student [4, 8, 11] 


986 


1,054 


1,278 


1,443 


4,761 


Implementation 


361 


374 


549 


461 


1,745 



August 9, 2004 



CHARACTERplus Implementation Project 
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Survey Factors 
(Outcomes) 

■ Students 

a Students’ Feelings of 
Belonging 

□ School Expectations 

□ Autonomy & 
influence 

□ Aitruism 

□ Feeiing of 
Competence 

□ Parent invoivement 



A - Autonomy 
B - Belonging 

C - Competence 

m Staff 

a Students’ Feelings 
of Beionging 

□ School 
Expectations 

□ Staff and Parent 
Reiations 

□ Staff Culture 

□ School Leadership 



August 9, 2004 



CHARACTERplus Implementation Project 
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Survey Factors (Outcomes) 



■ Parents 

□ Students’ Feelings 
of Belonging 

□ School 
Expectations 

a Parent & Staff 
Relations 

□ School Quality 

□ Parent Invoivement 



■ Impiementation 

□ Organized by the 
Eleven Principles of 
Effective Character 
Education 

□ Items based on the: 

■ Eleven Principles 

■ Ten Essentials 



August 9, 2004 



CHARACTERplus Implementation Project 
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Student Behavior Tracking Form 



■ Data collected for each student office 
referral: 

□ Date - Grade Level - Time of Incident 

□ Reason for referral based on 27 possible 
reasons divided into four categories: 

■ Acts Against Persons 

■ Acts Against Property 

■ Faiiure to Compiy 

■ Unauthorized Possession 



August 9, 2004 



CHARACTERplus Implementation Project 
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Key Barriers 1 



■ Obtaining and keeping school commitment. [This is 
ongoing.] 

■ Keeping schools on track - treatment schools 
participate in all treatment activities and control 
schools not implement the same or similar activities. 

■ IRB - obtain IRB approval to collect the data needed 
to successfully complete the research. 



August 9, 2004 



CHARACTERplus Implementation Project 
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Key Barriers 2 



■ Educating others of the difference between research 
and evaluation. Getting people to understand that 
the project is research-driven and that complete, 
valid data are critical to the project (unlike most 
program evaluations). [This is ongoing.] 

■ Educating participants in the value of research - 
that participation in the project is an opportunity, 
not only locally, but in the influence of state and 
national policy - that, as scientific-based research 
the results will be shared at the state and national 
levels and can impact future educational decisions. 
[This is ongoing]. 



August 9, 2004 



CHARACTERplus Implementation Project 
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Key Barriers 3 



■ Linking treatment specific outcomes to student 
achievement. State testing often changes; data 
measured in the spring are not available until the 
fall; data are in a format that may be statistically 
unfriendly; and typically available only at the school 
level. IRB requirements of active consent to obtain 
individual scores, adds to the problem. 

■ Collection of parent, staff, and student data in 64 
schools state-wide during February. 

■ Collection of student behavior data. 



August 9, 2004 



CHARACTERplus Implementation Project 
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Resistant Barriers 



■ Obtaining and keeping school commitment. 

■ Educating others of the difference between research 
and evaluation. Getting people to understand that 
the project is research-driven and that complete, 
valid data are critical to the project (unlike most 
program evaluations). 

■ Educating participants in the value of research - 
that participation in the project is an opportunity, not 
only locally, but in the influence of state and national 
policy - that, as scientific-based research the results 
will be shared at the state and national levels and 
can impact future educational decisions. 



August 9, 2004 



CHARACTERplus Implementation Project 
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Final Note 

This research project is a change paradigm 
for most of the participants. Participants 
are excited to be selected to participate in a 
large scale research project. However, it is 
difficult for them to internalize the 
importance they play in obtaining valid 
research data. Participants have not been 
part of research-driven experiences prior to 
this project 



August 9, 2004 



CHARACTERplus Implementation Project 



16 
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Alliance for Character Training (ACT) 




Alliance for Character 

T raining 

Christine Nardis 

Alliance City Schools 



Dr. Peter J. Leahy 

Institute for Health and Social Policy 
University of Akron 



CT Program History 



2001 SPEANS Planning Grant — Dept, of Health and 
Human Services 

■ Completed needs assessment (student, teacher, community, 
community leader surveys) 

■ Elicit community support and need for community backing 

■ Designed program and assessment mechanism during 2001- 
2002 school year 

2002 Partnerships in Character Education Grant 

■ Six month planning period (October 2002 — April 2003) 

■ Began implementation approximately in April 2003 (web 
page, radio); however, many activities did not begin until after 
this point (in-services, supplies in buildings, etc.) 




CT Program Description 

■ Character education program for grades K-8. 

■ Approximately 2,400 students in six buildings 

■ 4 elementary schools, one middle school, one K-8 
Catholic school 

■ Students from Alliance City Schools and Regina 
Coeli/St. Joseph Catholic School 

Secondary population-City of Alliance, Ohio 

■ Approximately 23,100 residents within Alliance 

■ Microcosm of a large city (dynamics) 

■ Highest crime rate per capita in the entire state 



Purpose of the Evaluation 




mmwwwm 



Monitor whether the “character education 
message” is being received by all staff in all 
schools 



llllllljii Measure changes in student attitudes toward 
school, moral principles, and risks as the 
program unfolds 

llllll^ Measure changes in student behaviors 

(attendance, discipline, reported crime rates) 

lll^ Raise community awareness of character issues 






Key Questions Evaluation was 
Designed to Answer 

Do attitudes improve more among 

students/ staff/ administrators in schools with ACT 

treatment versus schools without treatment? 

Do attimdes (of students/staff/administrators) 
improve more over time in treatment versus 
comparison schools? 

Do student behaviors change vis-a-vis prior trends and 
over time? (in comparison versus treatment schools) 




Key Questions Evaluation was 
Designed to Answer (continued) 




Does the community appear to be more 
knowledgeable about character issues by the 
conclusion of the program? 

JDo school staff perceive kids to have more 
positive attitudes? To exhibit more socially- 
approved behaviors? 

Do parents become more involved in their 
child’s school? 



Key Outcomes Measured in the 

Evaluation 



■ Student attitudes toward applications of 
character (ex. responsibility, caring) and 
perceptions of school 



J 

I 

J 



Student behaviors (grades, discipline, crime) 

Teacher/ Staff/ Administrator perceptions of 
student character and student behavior 




Teacher/ Staff/ Administrator perceptions of the 
penetration of character principles in the school 



Key Outcomes Measured in the 
Evaluation (^continued) 



■ Teacher attitudes (toward the profession) 



J 



Community (Alliance) attitudes toward character 
principles 

Teacher/ staff reports of parent school 
involvement 



Evaluation Design 




One group longitudinal analyses of student, 
teacher, staff, administrator and community 
attitude change. 

Longitudinal analysis of student behavior chang 



J 



Quasi-experimental comparison group “wait 
list” design. Two schools receive full 
implementation in Year 1. The remaining two 
schools enter full implementation in Year 3. 



Quasi-Experimental Study 



Building 


Treatment 


Staff # 


Enrollment 


Northside 

Elementary 


Year 1 


42 


448 


Parkway 

Elementary 


Year 1 


29 


236 


Regina Coeli/St. 
Joseph 


Year 1 


21 


221 


Rockhill 

Elementary 


Year 3 


43 


486 


South Lincoln 
Elementary 


Year 3 


53 


414 


*Alliance Middle 
School 


Year 1, but not 
part of this study 


100 


773 
































Quasi-Experimental Study: 
Student Surveys 



■ Student data collected each fall and subsequent 
spring on each entering fourth grade cohort 
(index grade) in all schools. 

■ 4* grade chosen because in the middle of age group 
(K-8) and readability levels are appropriate. 




Each spring thereafter survey is repeated as the 
youth move through the 5*^, 6*^*^ and grades. 

First baseline survey collected, 4’^’^ grade, Feb. 03. 

We are preparing to complete surveys this spring 
with the 4^^^ and grade students. 



Quasi-Experimental Study: 
Staffing and Enrollment 



Building 


Treatment 


Staff # 


Enrollment 


4*^ grade # 


5*^ grade # 


Northside 

Elem. 


Year 1 


42 


448 




F ““ ; 


Parkway 

Elem. 


Year 1 


29 


236 


m 'V 

c 


42 


Regina 

Coeli/St. 

Joseph 


Year 1 


21 


221 




26 


Rockhill 

Elem. 


Year 3 


43 


486 


^ 90 


77 


S. Lincoln 
Elem. 


Year 3 


53 


414 

1 


H /i:> 


43 




























Quasi-Experimental Design 
2003-2004 School Year Enrollment 



Treatment Group vs. Comparison Group 

905 

Students 

Norfhside Elem. Kockhill Elem. ^ 

Earkway Elem. South Eincoln Elem. 

Eegina Coelij St. Joseph 




Student Surveys 




J 



Survey-38 questions (see packet) 

Classroom teachers distribute surveys and 
envelopes to students and students return 
completed surveys in sealed, unmarked 
envelopes to better insure confidentiality in 
responses. 

Individual students are not identified or tracked. 

A passive consent letter is sent home to parents 
by the school superintendent prior to the survey. 



Student Survey Timeline 



Date of Student Survey 


Who will be surveyed? 


Spring 200 3 (February 18, 2003) 


All 4^^ grade students (baseline) 


Fall 2003 (October 21, 2003) 


All 4^^ grade students 


Spring 2004 (March 31 , 2004) 


All 4^^ and 5^^ grade students 


FaU 2004 


All 4^^ grade students 


Spring 2005 


All 4^^, 5^^, and 6^^ grade students 


FaU 2005 


All 4^^ grade students 


Spring 2006 


All 4^^, 5^^, 6^^, & 7^^ grade students 





















Figure 1 

Survey Administration by Schooi Year and Entry Cohort 




Teacher/ Staff Survey 



A survey of school staff, administrators, and 
teachers wiU be first administrated in spring 
2004 and executed yearly thereafter. Questions 
are from the CHARACTERp///r^'^ surveys from 
the St. Louis Cooperating School Districts, 
which are being used with the permission of Dr. 
Jon C. Marshall and Dr. Sarah Caldwell. 




Stark Poll 2003-Community Survey 




J 



A representative telephone survey of over 1400 
Stark County households conducted in May 
2003 by The University of Akron. 

Character questions measured citi2en 
perceptions of character issues and the 
importance of teaching character in the 
classroom. 






The same questions will be repeated in 2006. 



Additional Data 



■ EMIS information 

■ Crime statistics (local law enforcement statistics) 
mUB Average Daily Traffic (ADT) reports 

■ BiUboard exposures 

Newspaper data and circulation information 

||P Radio statistics (PSAs /CHARACTER 
COUNTS! character commentaries) 



Assessment Obstacles 



■ Getting a ruling whether passive consent would be 
appropriate for students in this project. The University 
of Akron’s IRB approved this protocol. 

■ Timing of student surveys. We were not able to 
complete our first student baseline until February 2003. 
The state emphasis on proficiency testing makes both 
timing and length of the student survey and important 
issue. 

■ Currently we are preparing to complete the spring 
student surveys along with the 

teacher/ staff/ administrator surveys; however, have not 
come across any problems as of yet. 




Assessment Successes 



■ Teachers pre-tested, critiqued and helped 
finalize survey and provide continuing feedback 
every cycle that the survey is administered to 
students. 



J 



The superintendent is extremely supportive and 
instrumental in cooperating in passive consent 
permission letters. 
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An IRB Tale 
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An IRB Tale 



Orange County Department of 

Education 

Institute for Character Education 




Timeline of Development 



• Character Education Grant Awarded October 2002 

• IRB member recruitment letter February 14*^, 2003 
(sample) 

• IRB Registered March 6'*^, 2003 

• FWA Granted March 18^^ 2003 

• OCDE IRB #1 initial meeting April 17'^, 2003 (sample 
agenda) 

• Institute for Character Education developmental approval 
June 24‘h 2003 

• Consent letter and instrument approval July 31 , 2003 

• Next meeting April, 2004 




Keys to Development 



Legal help 

Communication with Jeff Rodamar 
Sample IRB Policies from Dr. Mo 
Sample protocol document 
Superintendent support 




Future Barriers 



Replacing Mennbers 
Increase in research 




other options 



Rent an IRB 
Partner with a university 
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Issues in Complex Evaluation Design 



U. S. Department of Education, 
Office of Safe and Drug Free 
Schools 




Caliber Associates, Inc. 

Association for Supervision and Curriculum Development 
Character Education Partnership 
March, 2004 



Context 



• NCLB and many federally-funded grants call 
for use of “scientifically-based research” to 
generate evidence of program impact and 
effectiveness 

• Historically, research efforts have not \ 
contributed significantly to the resol ution/pf\ 
pressing education problems 

• Misunderstandings and disagreements about 
what SBR is and whether it is an appropriate 
approach for evaluation in schools 

• Perceptions and reaiity of persistent barriers 
to conducting SBR in field settings such as 
schools — how often is it used? 



Generating Knowledge about Education 



Boruch, De Moya, & Synder (2001) report: 

84 evaluations and studies planned bv^p^ 
FY 2000: N 



in 



• 1 randomized controlled trial 

• 51 need assessments 

• 49 program implementation/monitoring 
evaluations 

• 15 non-randomized impact evaluations 



CurmulatEve number of articles* 



Wbffe flif munber cf articles abdtii frmls in eftier areas ofsaciat' 

science research steadily ^town, fbe numher in cclucdfi^n f £ has behind* 

6,000 



SrOOO 



4,000 



3,000 



2,000 



1,000 



0 





■■■■ Crfminoloqy 






™“ Social Policy 
^ Psyciiology 




***• Education 






1950 1955 1961 



1971 



1981 



1991 



Vear 

* Artidex about deiinLtc and posxlEilc nnetooruzed HdcLtnalx. 

SOURCEi RotHH Bomctin Doroch^dt Moyj, dnd O^kt^kditr, 200 J 



Designs Consistent with 
Scientifically-Based Research 




Randomized controlled trials 



Quasi-experimental designs with c 
matched comparison conditions 

Regression discontinuity with repeated 
measures pre and post intervention 



Single-subject designs — multiple 
baseline, treatment reversal, interrupted 
time series 




What are some of the most challenging 
issues regarding scientifically-based 
research? 




• Definitional issues 

• Philosophical issues 

• Pragmatic issues 



Issues of Definition 



What is scientifically-based education research? 
What is meant by: 

• “High quality” research 

• “Strong” evidence 

• “Rigorous” research 

What is the distinction between research and 
evaluation? Is it trivial within the context of SBR? 




Issues of Epistemology and Theory 



• What role does an evaluator’s world-view play in 
choosing particular evaluation designs? 



• Does an empiricist and a naturalist view impa 
effectiveness in the same way? 



• If evidence of impact is generated through one 

paradigm (the scientific method) what, if any, are .the 
benefits and consequences? ’ ■ 

• What is an appropriate sequence of building 
knowledge — where are we as a field? 

• How does SBR and implementation evaluation, 
program fidelity, process evaluation relate? 

• How does a program’s theory of action inform 
decisions about evaluation design? 




Pragmatic Issues 



* Approvals (school participation, instructional delivery 
staff, IRB, student assent and parental consent) 

- Impact of approval success on sampling, design, 
Instrumentation 

- Understanding protections set by FERPA and PP 

* Design Choices 

- Access to population 

- Approvals 

- Costs 

- Recognizing forces driving design choices... always begin 
with your evaluation question then factor in other parameters 

- What if randomized controlled trials just can’t be attained — Is 
It still worth doing? 

- Balance between local needs and national needs 

* Roie of Teacher 




Why? 



Why has education not adoptee 
more experimental approach to 
knowledge generation? 




Summary 



• Issues of definition 

• Issues of science and philosophy 

• Issues of application 

Each are at the heart of our ability to implement 
complex evaluation designs. Our gathering today and 
tomorrow will help inform the work of ED and CETAC 
as we create resources and support for grantees. 



What are your thoughts . . .share with your colleagues 



